Creating a tagset, lexicon and guesser for a French tagger

We earlier described two taggers for French, a statistical one and a constraint-based one. The two taggers have the same tokeniser and morphological analyser. In this paper, we describe aspects of this work concerned with the definition of the tagset, the building of the lexicon, derived from an existing two-level morphological analyser, and the definition of a lexical transducer for guessing unknown words. 1 Background We earlier described two taggers for French: the statistical one having an accuracy of 95–97 % and the constraint-based one 97–99 % (see (Chanod and Ta-panainen, 1994; Chanod and Tapanainen, 1995)). The disambiguation has been already described, and here we discuss the other stages of the process, namely the definition of the tagset, transforming a current lexicon into a new one and guessing the words that do not appear in the lexicon. Our lexicon is based on a finite-state transducer lexicon (Karttunen et al., 1992). We describe in this section criteria for selecting the tagset. The following is based on what we noticed to be useful during the developing the taggers. 2.1 The size of the tagset Our basic French morphological analyser was not originally designed for a (statistical) tagger and the number of different tag combinations it has is quite high. The size of the tagset is only 88. But because a word is typically associated with a sequence of tags, the number of different combinations is higher, 353 possible sequences for single French words. If we also consider words joined with clitics, the number of different combinations is much higher, namely 6525. A big tagset does not cause trouble for a constraint-based tagger because one can refer to a combination of tags as easily as to a single tag. For a statistical tagger however, a big tagset may be a major problem. We therefore used two principles for forming the tagset: (1) the tagset should not be big and (2) the tagset should not introduce distinctions that cannot be resolved at this level of analysis. 2.2 Verb tense and mood As distinctions that cannot be resolved at this level of analysis should be avoided, we do not have information about the tense of the verbs. Some of this information can be recovered later by performing another lexicon lookup after the analysis. Thus, if the verb tense is not ambiguous, we have not lost any information and, even if it is, a part-of-speech tagger …